A Survey on Partitioning Skew Diminishing Techniques in Hadoop MapReduce Environment

In the era of Big Data, it creates large size of structured and unstructured data. MapReduce is an effective tool for parallel data processing. One significant issue in practical MapReduce applications is data skew: the imbalance in the amount of data assigned to each task. This causes some tasks to take much longer to finish than others and can significantly impact performance. Parallel data processing is the spirit of the Apache’s Hadoop. Hadoop is an open source execution of Google’s MapReduce schedule. It has two components. Hadoop Distributed File System (HDFS) for storing data part and MapReduce for processing the data part. MapReduce has become a popular programming model for constructing data processing applications. While being widely used, existing MapReduce schedulers still suffer from an issue known as partitioning skew, where the output of map tasks is unevenly distributed among reduce tasks. It causes most of the tasks to take much longer to finish than other tasks and can significantly impact on performance. This paper reviews the types of partitioning skew and several skew diminishing techniques to solve these issues.